Liquid cooling high-density pluggable modules for a network element

ABSTRACT

A network element include one or more modules each supporting one or more pluggable modules; and a first manifold and a second manifold each configured to connect to a conduit associated with a coldplate, wherein one of the first manifold and the second manifold is an inlet manifold and the other is an outlet manifold for a cooling fluid that flows through the conduit to cool the one or more pluggable modules. The one or more pluggable modules can be each a pluggable optical module that is one of compliant to any of XFP, SFP, XENPAK, X2, CFP, CFP2, CFP4, CFP8, QSFP, QSFP+, QSFP28, OSFP, and QSFP-DD and have a housing that has dimensions similar to any of XFP, SFP, XENPAK, X2, CFP, CFP2, CFP4, CFP8, QSFP, QSFP+, QSFP28, OSFP, and QSFP-DD.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present disclosure is a continuation-in-part (CIP) of U.S. patentapplication Ser. No. 17/941,140, filed Sep. 9, 2022, which is a CIP ofU.S. patent application Ser. No. 17/071,073, filed Oct. 15, 2020, whichclaims priority to U.S. Provisional Patent Application No. 62/915,187,filed Oct. 15, 2019, and U.S. Provisional Patent Application No.63/039,936, filed Jun. 16, 2020, the contents of each of which areincorporated by reference in their entirety.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to networking hardware. Moreparticularly, the present disclosure relates to systems and methods forliquid cooling high-density network pluggable modules for use innetworking, storage, computing, and the like network elements, includingpluggable optical modules as well as other types of pluggable modulesthat may not include optical interfaces.

BACKGROUND OF THE DISCLOSURE

Networks, data centers, cloud computing, and the like continue to grow.Equipment manufacturers must continue to deliver substantial continuousreductions in per-bit metrics related to cost, space, and power.Telecommunication, data communication, high-performance computing, andthe like systems are typically deployed in physical hardware shelves,chassis, rack-mounted units (“pizza boxes”), cabinets, etc. that aremounted in racks or frames, freestanding, in cabinets, or the like. Forexample, typical racks or frames are either 19, 21, or 23 inches inpractice. Various standards associated with racks or frames aredescribed by Telecordia's GR-63-CORE, “NEBS Requirements: PhysicalProtection” (April 2012), European Telecoms Standards Institute (ETSI),American National Standard Institute (ANSI), etc. One downside to thecontinual improvement in the per-bit metrics is the increased heat,i.e., power dissipation and power density, and the corresponding coolingrequirements (such as specified in the NEBS standards, note NEBS standsfor Network Equipment-Building System). Even further, network operatorswant to deploy frames in data centers, telecom central offices, etc. asdensely as possible, even further limiting cooling techniques, i.e.,constraining airflow between the front and back

Indeed, rapid advancements in fiber optic technology have increasedtransfer rates from 10 GbE to 40/100 GbE and currently advancing to200/400 GbE and beyond. With the emergence of 100-400 GbE technologies,the creation of network architectures free from bandwidth constraintshas been made possible. In addition, the small form factor of emergingpluggable optics allows a significant increase in the number of portsper package (i.e., port density). Pluggable optical modules may bedefined by a Multi-Source Agreement (MSA), proprietary designs byindividual vendors, etc. Generally, pluggable optical modules areconfigured to plug into a module or pizza box in a network element toprovide optical interconnectivity. There are other types of modules thatmay include non-optical interconnectivity such as computing, routing,switching, etc. Those skilled in the art will recognize there arevarious types of modules that may be used in a network element, all ofwhich are contemplated herein as pluggable modules.

The performance and longevity of pluggable modules depend on the ambienttemperature they operate in and the thermal characteristics of thepackaging of these devices. Although most field-programmable gate arrays(FPGAs), application-specific integrated circuit (ASICS), and otherelectronic components may need to be maintained below ˜100° C., mostoptical components including pluggable optics may need to be maintainedat a temperature below ˜70° C. to ensure reliable transmission of data.The ever-increasing port density coupled with the low-temperatureratings associated with optical modules makes the thermal management ofpluggable optics more and more challenging. Also, these modules aremeant to be inserted and removed, so there are further physicalchallenges for cooling related to pluggable modules.

From an architectural perspective, the real estate in a 1 Rack Unit(1RU) faceplate of a network element is fixed. To increase the datatransfer rate, more pluggable optics and faster pluggable optics need tobe fit in a faceplate. For example, the industry target now is to fit36xQSFPs in a 1RU faceplate (i.e., QSFP stands for Quad Small Formfactor Pluggable and includes QSFP28 for 100 GbE, QSFP56 for 200 GbE,QSFP-DD (Double Density) for 400 GbE), while current products may fit24xQSFP28 in a 1RU platform (aka. pizza box). Moving from 24xQSFP28 (100GbE—heat dissipation: 3.5 W per pluggable module) to 36xQSFP-DD (400GbE—heat dissipation: 24 W per pluggable module) creates a substantialthermal challenge with the present technologies. Increasing port densityis an industry trend, and the abovementioned numbers are likely toincrease in the future. To keep up with the trend, improved cooling isneeded.

BRIEF SUMMARY OF THE DISCLOSURE

The present disclosure relates to systems and methods for liquid coolinghigh-density network pluggable modules for use in networking, storage,computing, and the like network elements, including pluggable opticalmodules as well as other types of pluggable modules that may not includeoptical interfaces. As described herein, pluggable modules can also bereferred to as a plug. The present disclosure includes variousembodiments of pluggable modules with liquid cooling approaches thatprovide substantial cooling enhancement relative to conventional aircooling approaches. This enables significantly higher power plugs can becooled through the elimination of a dry contact interface and employmentof liquid cooling. In various embodiments, the present disclosureincludes various embodiments of liquid cooling approaches for pluggablemodules. In an embodiment, the present disclosure includes a ridingliquid coldpate with a coating on a housing of a pluggable module. Thisapproach requires no changes to the pluggable module, which isadvantageous in terms of compliance. In another embodiment, the presentdisclosure includes a common liquid coldpate shared amongst a pluralityof pluggable modules each having a coating or plating on their housing.This approach also requires no changes and is useful where a networkelement has a large number of pluggable modules in a row. In a furtherembodiment, the present disclosure includes an integrated liquid-coolingapproach that is integrated into the body of a pluggable module. Thisapproach provides improved performance over the coldplate approaches butrequires changes to the housing, i.e., updated MSA standardization. Inyet another embodiment, the present disclosure includes liquid-cooledsled modules. Sled modules can provide additional functionality insteadof just optical interconnections, e.g., computing, routing, switching,etc.

In an embodiment, a network element includes one or more modules eachsupporting one or more pluggable modules; and a first manifold and asecond manifold each configured to connect to a conduit associated witha coldplate, wherein one of the first manifold and the second manifoldis an inlet manifold and the other is an outlet manifold for a coolingfluid that flows through the conduit to cool the one or more pluggablemodules. The one or more pluggable modules can be each a pluggableoptical module that is one of compliant to any of XFP, SFP, XENPAK, X2,CFP, CFP2, CFP4, CFP8, QSFP, QSFP+, QSFP28, OSFP, and QSFP-DD and have ahousing that has dimensions similar to any of XFP, SFP, XENPAK, X2, CFP,CFP2, CFP4, CFP8, QSFP, QSFP+, QSFP28, OSFP, and QSFP-DD.

The first manifold and the second manifold can be in a stackedconfiguration. The one or more modules can support the one or morepluggable modules in a plurality of rows, and the network elementincludes the first manifold and the second manifold separately for eachof the plurality of rows. The one or more pluggable modules can includethe coldplate one of integrated into a housing, immersed cooling, anddirect processor cooling. The cooling fluid can include an electricallyinert fluid that is either a single or two-phase flow. The one or morepluggable modules can each include fluid connectors configured toconnect to the manifolds and electrical connectors. The fluid connectorscan be configured to align the electrical connectors. The coldplate canbe a riding coldplate that is configured to make thermal contact withthe one or more pluggable modules once inserted. The network element canfurther include a spring configured to bias the one or more pluggablemodules to the coldplate. The coldplate can include a thermal contactwith a coating thereon. The conduit can include one or more bends/turnsto form a bending, winding, tortuous, serpentine, or otherwisecircuitous path. The coldplate can include fluid connectors that areconfigured to blind mate and quickly disconnect. The network element canfurther include leak mitigation. The network element can further includea sensor configured to detect leakage of the cooling fluid in thenetwork element.

In another embodiment, a method includes providing a network elementcomprising one or more modules each supporting one or more pluggablemodules; and a first manifold and a second manifold each configured toconnect to a conduit associated with a coldplate, wherein one of thefirst manifold and the second manifold is an inlet manifold and theother is an outlet manifold for a cooling fluid that flows through theconduit to cool the one or more pluggable modules.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated and described herein withreference to the various drawings, in which like reference numbers areused to denote like system components/method steps, as appropriate, andin which:

FIGS. 1-4 are various perspective diagrams of a platform. FIG. 1 is afront perspective diagram of the platform, FIG. 2 is a front view of theplatform, FIG. 3 is a rear perspective diagram of the platform, and FIG.4 is an internal cross-sectional diagram of the platform.

FIGS. 5-7 are various perspective diagrams of an array of a module cageswith coldplates. FIG. 5 is a perspective diagram of the module array.FIG. 6 is a top perspective diagram of a portion of the module array ofFIG. 5 . FIG. 7 is a side perspective diagram of the module cage andcoldplate from the array of FIG. 5 .

FIGS. 8 and 9 are perspective diagrams of a cage and module of themodule of FIGS. 5-7 with a coldplate.

FIGS. 10 and 11 are perspective diagrams of the coldplate of FIGS. 5-9 .

FIG. 12 is a perspective diagram of a coldplate.

FIG. 13 is a perspective diagram of a coldplate.

FIG. 14 is a perspective diagram of a coldplate.

FIG. 15 is a perspective diagram of the module array with multiple rowsof cages and coldplates.

FIG. 16 is a side perspective diagram of the module array of FIG. 15 .

FIG. 17 is a perspective diagram of a pluggable optics module with anintegrated coldplate. FIG. 18 is an exploded perspective diagram of thepluggable optics module with the integrated coldplate of FIG. 17 . FIG.19 is a side perspective diagram of the pluggable optics module with theintegrated coldplate of FIG. 17 . FIG. 20 is a back perspective diagramof the pluggable optics module with the integrated coldplate of FIG. 17. FIG. 21 is a perspective diagram of the integrated coldplate of FIGS.17-20 without a top surface thereof.

FIG. 22 is a top perspective diagram of a module with the pluggableoptics module of FIGS. 17-20 in a cage. FIG. 23 is a side perspectivediagram of the module of FIG. 22 . FIG. 24 is a side perspective diagramof the module of FIGS. 22 and 23 without the cage. FIG. 25 is aperspective diagram of a portion of the module and cage of FIGS. 22-24 .

FIG. 26 is a side perspective diagram of an embodiment of the modulewithout the cage.

FIG. 27 is a schematic diagram of the connection zones of the module ofFIGS. 22-26 .

FIG. 28 is a schematic diagram of a leak mitigation system for themodule.

FIGS. 29 and 30 are perspective diagrams of an embodiment of a sledmodule.

FIGS. 31-34 are perspective diagrams of an embodiment of a systemsupporting 36 or 54 pluggable optics module, such as QSFP-DD modules.

FIG. 35 is a graph illustrating temperature for different flow rates forthe liquid cooling using a coldplate and for air cooling with a heatsink.

FIGS. 36 and 37 are various diagrams of a module that supports multiplepluggable modules with a common coldplate. FIG. 36 is a perspectivediagram of the module, and FIG. 37 is a side perspective diagram of themodule of FIG. 36 .

FIGS. 38-39 are various perspective diagrams of another network elementplatform. FIG. 38 is a front perspective diagram of the network elementplatform, and FIG. 39 is a rear perspective diagram of the networkelement platform.

FIGS. 40-41 are perspective diagrams of a circuit pack without a vaporchamber (FIG. 40 ) to show QSFP-DD heatsinks and with the vapor chamber(FIG. 41 ).

FIGS. 42-43 are perspective diagrams of the circuit pack without a vaporchamber (FIG. 42 ) to show liquid cooling cold blocks to replace theQSFP-DD heatsinks and with the vapor chamber (FIG. 43 ).

FIG. 44 is a diagram of the backplane and the liquid cooling cold blockswithout other components in the shelf.

FIG. 45 is a front perspective diagram and FIG. 46 is a rear perspectivediagram of a coolant distribution manifold box connected to the liquidcooling cold blocks with the backplane removed for illustrationpurposes.

FIGS. 47-50 are various perspective diagrams of a liquid cooling coldblock. FIG. 47 is a top perspective diagram of the liquid cooling coldblock. FIG. 48 is a bottom perspective diagram of the liquid coolingcold block. FIG. 49 is a bottom perspective diagram of the liquidcooling cold block with cooling lines removed. FIG. 50 is a top diagramof the liquid cooling cold block with cooling lines connected to therear manifold.

FIGS. 51-52 are a front perspective view (FIG. 51 ) and a rearperspective view of a coolant distribution manifold box.

DETAILED DESCRIPTION OF THE DISCLOSURE

Again, the present disclosure relates to systems and methods for liquidcooling high-density network pluggable modules for use in networking,storage, computing, and the like network elements, including pluggableoptical modules as well as other types of pluggable modules that may notinclude optical interfaces. As described herein, pluggable modules canalso be referred to as a plug. The present disclosure includes variousembodiments of pluggable modules with liquid cooling approaches thatprovide substantial cooling enhancement relative to conventional aircooling approaches. This enables significantly higher power plugs can becooled through the elimination of a dry contact interface and employmentof liquid cooling. In various embodiments, the present disclosureincludes various embodiments of liquid cooling approaches for pluggablemodules. In an embodiment, the present disclosure includes a ridingliquid coldpate with a coating on a housing of a pluggable module. Thisapproach requires no changes to the pluggable module, which isadvantageous in terms of compliance. In another embodiment, the presentdisclosure includes a common liquid coldpate shared amongst a pluralityof pluggable modules each having a coating on their housing. Thisapproach also requires no changes and is useful where a network elementhas a large number of pluggable modules in a row. In a furtherembodiment, the present disclosure includes an integrated liquid-coolingapproach that is integrated into the body of a pluggable module. Thisapproach provides improved performance over the coldplate approaches,but requires changes to the housing, i.e., updated MSA standardization.In yet another embodiment, the present disclosure includes liquid-cooledsled modules. Sled modules can provide additional functionality insteadof just optical interconnections, e.g., computing, routing, switching,etc.

In an embodiment, the present disclosure uses a module for multiplepluggable optics held by cages connected to a PCB. One or morecoldplates can be positioned to thermally contact the pluggable opticsto liquid-cool the pluggable optics. The coldplates can be separate fromthe pluggable optics with contact pressure between the pluggable opticsand the coldplate, or the liquid cooling can be integrated into thepluggable optics, which removes the necessity to have a dry slidinginterface between the coldplate and the pluggable optics that canincrease thermal resistance. That is, the coldplate is an externaladd-on to a housing with the coldplate, including liquid cooling to ametal structure that can cool an adjacent housing. The integrated liquidcooling is built directly into a housing of the pluggable module. Due tothe thermal properties of liquid solutions, even at low flow rates, theoperating temperature of the liquid-cooled pluggable optics can be lowerthan the operating temperature of air-cooled pluggable optics in anideal air-cooling condition. Further, with superior thermal propertiesof liquids, the space required for cooling can be minimized, which opensthe door for more compact architectures, i.e., better than heat sinksand air cooling.

The coldplates are each connected to an inlet manifold and an exhaustmanifold. Each of the coldplates, the inlet manifold, and the exhaustmanifold can include connecting tubes. For the integrated cooling, theconnecting tubes can be integral to the pluggable optics. For pluggableoptics with integrated cooling, the connecting tubes can include blindmate quick connections/releases (liquid quick-disconnects) to allow forthe pluggable optics to connect to the inlet and exhaust manifoldswithout allowing the fluid therein to leak. The connecting tubes for theriding coldplates or the connecting tubes for the inlet and exhaustmanifolds can include integral flanges for connecting to the other ofthe connecting tubes for the coldplates or the connecting tubes for theinlet and exhaust manifolds. Such a configuration simplifies theconnection between the coldplates and the manifolds (inlet and exhaust)and eliminates the need for flyover cables, separate connectors,separate fittings, and the like, which can reduce the cost ofmanufacturing and can reduce the number of potential locations forleaks.

For the pluggable optics with integrated cooling, a connector of themodule can include a common body that integrates both the liquid andelectrical connections. The connector is configured to connect to bothelectrical and liquid connections of the pluggable optic simultaneously.Furthermore, the liquid quick-disconnects can act as alignment pins(removing the need for separate alignment pins) and can help controltolerance between the liquid and electrical connections.

In embodiments, the pluggable optics with integrated cooling can includeone or more of an integrated coldplate, immersion cooling withdielectric fluid, direct processor liquid cooling with a dielectricfluid, liquid cooling for the electrical connections, and cooling to andbeyond the faceplate. With such cooling, more power can be delivered tothe pluggable optics allowing for an increase in the pluggable opticsoperating power and providing a decrease in the touch temperature of thecase.

Also, the present disclosure utilizes various terms in the art such asmodule and card. Those of ordinary skill in the art will recognize theseterms may be used interchangeably. Further, these do not requireremovability. That is, a module or card may be fixed in a platform. Onthe other hand, an interface card or a pluggable module (again, theseterms may be used interchangeably) is selectively removable from a cage,slot, etc. in the platform, including in a module or card. Also, theterm platform is used herein to denote a hardware device housing themodules or cards. The platform may include a shelf, chassis,rack-mounted unit, “pizza box,” etc.

Example Network Element

FIGS. 1-4 are various perspective diagrams of a network element 100.FIG. 1 is a front perspective diagram of the network element 100, FIG. 2is a front view of the network element 100, FIG. 3 is a rear perspectivediagram of the network element 100, and FIG. 4 is an internalcross-sectional diagram of the network element 100. FIGS. 1-4 are fromcommonly-assigned U.S. Pat. No. 9,769,959, issued Sep. 19, 2017, andentitled “HIGH DENSITY NETWORKING SHELF AND SYSTEM,” the contents ofwhich are incorporated herein by reference. The network element 100 canbe a shelf, a system, a chassis, etc. forming a network element, a node,etc. in a network. The network element 100 can include front and rearair intake/exhaust without side ventilation, thereby maintaining NEBScompliance. Additionally, the network element 100 is a half-rack systemthat is scalable to a double (full rack) sized system. The networkelement 100 is presented as an example for illustration purposes. Thoseskilled in the art will recognize other physical embodiments arecontemplated. That is, the present disclosure contemplates use with anyhardware platform having pluggable modules.

In an embodiment, the network element 100 can be a network element thatmay consolidate the functionality of a Multi-Service ProvisioningPlatform (MSPP), Digital Cross-Connect (DCS), Ethernet and/or OpticalTransport Network (OTN) switch, Dense Wave Division Multiplexing (DWDM)platform, etc. into a single, high-capacity intelligent switching systemproviding Layer 0, 1, and 2 consolidation. In another exemplaryembodiment, the network element 100 can be any of an OTN Add/DropMultiplexer (ADM), a SONET/SDH/OTN ADM, an MSPP, a DCS, an opticalcross-connect, an optical switch, a router, a switch, a DWDM terminal,wireless backhaul terminal, an access/aggregation device, etc. That is,the network element 100 can be any digital and/or optical system withingress and egress signals and switching therebetween of channels,timeslots, tributary units, packets, etc. utilizing OTN, SONET, SDH,Ethernet, IP, etc. In another embodiment, the network element 100 can bea high-rate Ethernet switch or router. In a further embodiment, thenetwork element 100 can be a DWDM terminal. In yet another embodiment,the network element 100 can be a compute, wireless, storage, or anothertype of hardware platform. The key aspect of the network element 100with the present disclosure and any other platform are the pluggablemodules, via interface cards 114 in a cage.

The network element 100 includes a housing 102, which can refer to anyshelf, rack, cabinet, case, frame, chassis, or other apparatus used toarrange and/or support a plurality of electronic/optical components suchas removable cards, including modules with interface cards 114 andswitch fabric cards 116. The housing 102 may be metal, plastic, orcombination, or other suitable material and similar in construction toother housings, cabinets and/or racks used to hold electronic/opticalcomponents in place. Further, the housing 102 may be rack mounted in anETSI, ANSI, etc. compliant rack or frame, as well as being deployed in acabinet, etc. The housing 102 has a front side 104, a rear side 106opposite the front side 104, a right side 108 adjacent to both the frontside 104 and the rear side 106, and a left side 110 opposite the rightside and adjacent to both the front side 104 and the rear side 106.Airflow in the network element 100 is between the front side 104 and therear side 106; there may be no airflow through or between the sides 108,110. Of course, other embodiments may include side-to-side airflow,top-to-bottom airflow, as well as combinations of different directions.

The housing 102 supports a set of interface cards 114 and, optionally, aset of switch fabric cards 116. The interface cards 114 are arranged ina first direction 120. The switch fabric cards 116 are arrangedsubstantially orthogonally, i.e., perpendicular, to the first direction120. In this embodiment, the interface cards 114 are vertically aligned,and the switch fabric cards 116 are horizontally aligned. The cards 114,116 may optionally be surrounded by a separate metallic Faraday Cage,including, for example, a metal mesh screen. The orthogonal arrangementof the switch fabric cards 116 as compared with the interface cards 114can form the recessed portion 40 as described herein.

The interface cards 114 can include selectively inserted pluggableoptical modules with the liquid cooling approaches described herein.Again, the interface cards 114 can be referred to as line cards, lineblades, I/O modules, etc. and can include a plurality of optical modulesin the front. For example, the optical modules can be pluggable modulessuch as, without limitation, XFP, SFP, XENPAK, X2, CFP, CFP2, CFP4,CFP8, QSFP, QSFP+, QSFP28, OSFP, QSFP-DD, etc. XFP stands for 10 Gigabitsmall Form factor Pluggable. SFP stands for Small Form factor Pluggable.CFP stands for C Form factor Pluggable, and includes variants such asCFP2, CFP4, CFP8, CFP2-DCO (Digital Coherent Optics), etc. OSFP standsfor Octal Small Form factor Pluggable. Further, the interface cards 114can include a plurality of optical connections per module. The interfacecards 114 can include wavelength division multiplexing interfaces,short-reach interfaces, and the like, and can connect to other interfacecards 114 on remote network elements, end clients, edge routers, and thelike.

From a logical perspective, the interface cards 114 provide ingress andegress ports to the network element 100, and each interface card 114 caninclude one or more physical ports. The optional switch fabric cards 116are configured to switch channels, timeslots, tributary units, packets,cells, etc. between the interface cards 114. The interface cards 114and/or the switch fabric cards 116 can include redundancy as well, suchas 1:1, 1:N, etc. In an embodiment, the high-density network element 100can be 15-16RU with 12 slots for line modules housing the interfacecards 114 and 4 slots for the switch fabric cards 116. Here, thehigh-density network element 100 can dissipate between 600-750 W.Further, the switch fabric cards 116 can be single fabric or doublefabric (with additional pins to the backplane from the single fabric).Additionally, the network element 100 contemplates operation in an ETSI,ANSI, 19″, or 23″ rack or frame.

The network element 100 can include common equipment 130, powerconnections 132, and a fiber manager 134. The common equipment 130 isutilized for Operations, Administration, Maintenance, and Provisioning(OAM&P) access; user interface ports; and the like. The network element100 can include an interface for communicatively coupling the commonequipment 130, the interface cards 114, and the switch fabric cards 116therebetween. For example, the interface can be a backplane, midplane, abus, optical or electrical connectors, or the like. The interface cards114 are configured to provide ingress and egress to the network element100.

Those of ordinary skill in the art will recognize the network element100 can include other components which are omitted for illustrationpurposes, and that the systems and methods described herein arecontemplated for use with a plurality of different network elements withthe network element 100 presented as an example type of network deviceor hardware platform. For the high-density network element 100, otherarchitectures providing ingress, egress, and switching therebetween arealso contemplated for the systems and methods described herein. Those ofordinary skill in the art will recognize the systems and methods can beused for practically any type of network device, which includesinterface cards 114 requiring significant cooling. For example, thepresent disclosure also contemplates a network element as an integratedrack unit, commonly referred to as a pizza box. The integrated rack unitmay be 1-2RU in height and only include removable interface cards 114.Of note, the integrated rack unit has significantly less space forcooling relative to the network element 100. However, those skilled inthe art should recognize the liquid cooling approaches described hereinfor pluggable modules are not limited to any specific implementation ofthe network element.

The network element 100 includes a housing with a front side, a rearside opposite the front side, a right side adjacent to both the frontside and the rear side, and a left side opposite the right side andadjacent to both the front side and the rear side. One or more modulesin the housing each including a plurality of cages supporting removableinterface cards. The removable interface cards can be pluggable opticalmodules, sled modules, etc., which require cooling.

Riding Liquid Coldplate and Coating

FIGS. 5-7 are various perspective diagrams of a module 200 that supportsmultiple pluggable modules 10. FIG. 5 is a perspective diagram of themodule 200. FIG. 6 is a top perspective diagram of a portion of themodule 200 of FIG. 5 . FIG. 7 is a side perspective diagram of themodule 200 of FIG. 5 . FIGS. 8 and 9 are perspective diagrams of a cage220 and the pluggable module 10 that may be inserted in the module 200of FIGS. 5-7 with a coldplate 230. FIGS. 10 and 11 are perspectivediagrams of the coldplate 230 of FIGS. 5-9 .

Specifically, the embodiment described herein relates to the coldplate230, which is a riding coldplate, namely, it is not physically attachedto the pluggable module 10. The pluggable module 10 can include acoating or a plating on its housing, or the coldplate can include thecoating on its housing. The advantage of this approach is it does notrequire hardware changes to the pluggable module 10, and the coldplate230 can be integrated in a corresponding cage or the like in the networkelement. The pluggable module 10 can be any of XFP, SFP, XENPAK, X2,CFP, CFP2, CFP4, QSFP, QSFP+, QSFP28, OSFP, QSFP-DD, and the like.

As can be seen in FIGS. 5-7 , the module 200 can include a faceplate210, a Printed Circuit Board (PCB) 290, multiple cages 220, and a heatexchange assembly. The heat exchange assembly includes a first manifold250, a second manifold 260, and one or more coldplates 230 for coolingpluggable optics inserted into the cages 220. Also, this embodimentcontemplates a network element with multiple cages 220 and a singleliquid cooling system via the manifolds 250, 260.

The PCB 290 can be connected to the faceplate 210. The multiple cages220 can be in a single row configuration or can be configured inmultiple rows as shown in FIGS. 15 and 16 . Each row of multiple cages220 can include a separate heat exchange assembly.

The first manifold 250 can include a chamber 252 and multiple connectingducts 255, and the second manifold 260 can include a chamber 262 andmultiple connecting ducts 265. The chambers 252 and 262 branch into themultiple connecting ducts 255 and 265, respectively. The multipleconnecting ducts 255 are integral to the chamber 252, and the multipleconnecting ducts 265 are integral to the chamber 262. By being integral,the multiple connecting ducts 255 and 265 can be integrally formed with,metallurgically bonded with or otherwise permanently joined to thechambers 252 and 262, respectively. One of the first manifold 250 andthe second manifold 260 is an inlet manifold for supply cooling liquidor fluid to the coldplates 230, while the other is an exhaust manifoldfor removing the cooling liquid from the coldplates 230. Generally, thetechniques described herein use a fluid which can include a coolingliquid as well as other embodiments such as a two-phase liquid and gasflow. The first manifold 250 and the second manifold 260 can be in fluidcommunication with a reservoir for the cooling liquid and a pump. Thecooling liquid can be a solution, such as propylene glycol solution,other solutions used for electronic cooling, and the like.

Referring to FIGS. 5-9 , each coldplate 230 can include a plate 231, afirst connector 235, a second connector 236, and a conduit 232. Theplate 231 includes a mating surface 239 configured to be in thermalcontact with an interface surface of the pluggable module 10 insertedinto the corresponding cage 220. The plate 231 can be spring-loaded tothe cage (i.e. using a spring clip) to create enough contact pressure atthe interface of pluggable module and the coldplate 230 to minimize thethermal resistance and improve the heat flow path. The plate 231 at themating surface 239 and the corresponding surface of the pluggable module10 can include a coating thereon, such as a coating that improves thethermal resistance caused by the dry/sliding interface. The coating caneliminate small air gaps at the interface, which reduces thermalresistance. The coating can include surface treatment techniques used toreduce the thermal resistance at the dry/sliding interface. As describedherein and as is known in the art, thermal contact means there areconnection points between the pluggable module 10 and the coldplate 230for the flow of heat, namely from the pluggable module 10 to thecoldplate 230. The coating is used because the pluggable module 10 isremovable from the cage 220.

The first connector 235 and the second connector 236 can be integral tothe conduit 232 and can be integral to the plate 231. By being integral,the first connector 235 and the second connector 236 can be integrallyformed with, metallurgically bonded with or otherwise permanently joinedto the conduit 232 and/or the plate 231.

One of the first connector 235 and the corresponding connecting duct 255can include an integral flange 245 that is integrally formed with,metallurgically bonded with or otherwise permanently joined to the oneof the first connector 235 or corresponding connecting duct 255. Theflange 245 is configured to directly receive the other of the firstconnector 235 and the corresponding connecting duct 255 for joining thefirst connector 235 to the corresponding connecting duct 255.

One of the second connector 236 and the corresponding connecting duct265 can include an integral flange 246 that is integrally formed with,metallurgically bonded with or otherwise permanently joined to the oneof the second connector 236 or corresponding connecting duct 265. Theflange 246 is configured to directly receive the other of the secondconnector 236 and the corresponding connecting duct 265 for joining thesecond connector 236 to the corresponding connecting duct 265.

Due to the flanges 245 and 246 being integral to either the first andsecond connectors 235, 236 or to the corresponding connecting ducts 255,265, the first connector 235 and the corresponding connecting duct 255can be directly connected without any connectors and fittings, and thesecond connector 236 and the corresponding connecting duct 265 can bedirectly connected without connectors and fittings. These directconnections can simplify the heat exchange assembly and reduce assemblycosts and potential leakage locations of the heat exchange assembly.

The first connector 235, the second connector 236, and/or thecorresponding connecting ducts 255, 265 can be small tubes/pipes and canbe made of a flexible material to accommodate the mobility of coldplates230 when optics are plugged in and to accommodate any variance indimensions of the first connector 235, the second connector 236, and thecorresponding connecting ducts 255, 265 for forming the connections. Inanother embodiment, the first connector 235 and the flange 245 can beembedded copper tubes. The connections between the first connector 235,the second connector 236, and the corresponding connecting ducts 255,265 can be permanently mated, which can reduce costs.

At least one of the first connector 235, the second connector 236, andthe corresponding connecting ducts 255, 265 can include a singletube/pipe with multiple bends/elbows. The first manifold 250 and thesecond manifold 265 can be in a stacked configuration. The multiplebends/elbows allow the connection between the coldplate 230 and firstand second manifolds 250, 260 that is offset from the coldplate to beformed without further flyover cables/connectors/fittings.

FIGS. 10-14 are perspective diagrams of a coldplate 230. Referring toFIGS. the conduit 232 is configured to cool the plate 231 with coolingliquid flowing therethrough. The conduit 232 can include one or morebends/turns to form a bending, winding, tortuous, serpentine, orotherwise circuitous path to ensure enough convection between thecoldplate 230 and the cooling fluid to sufficiently cool the pluggableoptics. Of note, the coldplate 230 can have different dimensions, shape,and structure of the conduit 232 based on the application and theunderlying module that it is used with.

The conduit 232 can be a pipe, such as a copper pipe, that ismetallurgically bonded to the plate 231, such as by soldering, can beformed directly in the plate 231, and the like. The conduit 232 can beflattened, which can reduce a height of the module 200 and can increasea thermal contact area between the conduit 232 and the plate 231, andthus, reduce thermal resistance.

The conduit 232 can be subdivided into multiple flow paths 233 for anentirety of the flow through the coldplate 230, can be subdivided intomultiple flow paths 233 in sections of the coldplate 230, such asbetween bends, and the like.

As can be seen in FIG. 11 , in some embodiments, the coldplate 230includes a pedestal 237 that protrudes from the plate 231 and forms adry/sliding interface between the pluggable module 10 and the coldplate230. In embodiments, a thin coated layer is applied to an outer surfaceof the pedestal 237. Again, the thin coated layer can include surfacetreatment techniques used to reduce the thermal resistance at thedry/sliding interface.

FIG. 15 is a perspective diagram of the module 200 with multiple rows ofcages 220 and coldplates 230. FIG. 16 is a side perspective diagram ofthe module 200 of FIG. 15 . As noted above, each row of cages 220 caninclude separate heat exchange assemblies, with separate first andsecond manifolds 250, 260 and associated connections. Alternatively, asingle set of first and second manifolds 250, 260 can be used, which areconfigured to connect to both rows of coldplates 230. This embodimentcontemplates use in a network element supporting a large number ofpluggable modules 10.

Of note, the riding coldplate approach advantageously allows use ofliquid cooling techniques with any existing pluggable module 10, i.e.,no change needed for compliance. This is because the coldplate 230 isexternal and makes thermal contact with the pluggable module 10. Thatis, the cage 220 includes the coldplate 230 instead of any heatsink.

Integrated Coldplate

FIG. 17 is a perspective diagram of a pluggable module 320 with anintegrated coldplate 330. FIG. 18 is an exploded perspective diagram ofthe pluggable module 320 with the integrated coldplate 330 of FIG. 17 .FIG. 19 is a side perspective diagram of the pluggable module 320 withthe integrated coldplate 330 of FIG. 17 . FIG. 20 is a back perspectivediagram of the pluggable module 320 with the integrated coldplate 330 ofFIG. 17 . In embodiments, the pluggable module 320 (plug) is ahigh-density network pluggable optical module. That is, the pluggablemodule 320 can be similar to any of XFP, SFP, XENPAK, X2, CFP, CFP2,CFP4, QSFP, QSFP+, QSFP28, OSFP, QSFP-DD, and the like, but may requirea new standard designation due to the housing change in the integrationof the coldplate 330, e.g., QSFP-DD-LC where LC is Liquid Cooled. Ofcourse, any other designation may be used, but the point to note isintegration requires modification to the housing of the pluggable module10, which requires new standardization.

Referring to FIGS. 17-20 , the plug 320 includes a main body 321,circuitry 322, an integrated coldplate 330, a first plug fluid connector335 and a second plug fluid connector 336. The circuitry 322, includingthe plug PCB, the processor, and the like, of the plug 320 is held bythe main body 321. The integrated coldplate 330 connects to the mainbody 321 and is configured to provide cooling for the circuitry 322. Inthe embodiment illustrated, the integrated coldplate 330 acts as a coverfor the plug 320 covering at least a portion of the circuitry 322 heldwithin the main body 321. The plug 320 in FIGS. 17 and 18 is aQSFP-DD-like module with integrated liquid cooling. The plug 320 inFIGS. 19 and 20 is an OSFP-like module with integrated liquid cooling.

In some embodiments, the integrated coldplate 330 includes immersioncooling for the electrical components where the central region 328 ofthe plug 320 is filled with an electrically inert fluid. In oneembodiment, the integrated coldplate 330 is configured to remove theheat from the electrically inert fluid. In another embodiment, theintegrated coldplate 330 includes one or more flow paths through thecentral region 328 to directly cool the circuitry 322 with theelectrically inert fluid flowing through the central region 328.

In some embodiments, the integrated coldplate 330 is configured to coolthe processor indirectly with a surface contacting the processor ordirectly by interfacing with and providing cooling fluid throughchannels formed in the processor.

In embodiments, sides of the main body 321 extend upward beyond thecircuitry 322 and provide support and mate with the integrated coldplate330. As can be seen in FIG. 20 , the integrated coldplate 330 caninclude a plate 331 and protrusions 333 extending down from the plate331 into the main body 321. The protrusions 333 are oriented to positionthe integrated coldplate 330 relative to the main body 321 and can beused to secure the integrated coldplate 330 to the main body 321.

The plug 320 also includes a plug connector 329 that electronicallyconnects the plug 320, and in particular, the circuitry 322, to themodule that receives the plug 320. The plug connector 329 can bepositioned at a back end of the plug PCB and can be part of the plugPCB. In embodiments, the integrated coldplate 330 is configured todirectly cool one or more of the plug connector 329 and the connector ofthe module that is configured to receive the plug connector 329.

FIG. 21 is a perspective diagram of the integrated coldplate 330 ofFIGS. 17-19 without a top surface thereof. Referring to FIG. 21 , theintegrated coldplate 330 includes a conduit 332. The conduit 332 caninclude one or more bends/turns to form a bending, winding, tortuous,serpentine, or otherwise circuitous path to ensure enough convectionbetween a body of the integrated coldplate 330 and the cooling fluid tosufficiently cool the plug 320 and the circuitry 322 therein. Theconduit 332 can be subdivided into multiple flow paths for an entiretyof the flow through the integrated coldplate 330, can be subdivided intomultiple flow paths in sections of the integrated coldplate 330, such asbetween bends, and the like. In embodiments, the conduit 332 includesone of the conduits 232 illustrated in FIGS. 10, 12, and 13 .

The first plug fluid connector 335 and the second plug fluid connector336 are in fluid communication with opposite ends of the conduit 332,where one acts as an inlet and the other acts as an outlet for theconduit 332.

FIG. 22 is a top perspective diagram of a module 300 with the pluggablemodule 320 of FIGS. 17-20 . FIG. 23 is a side perspective diagram of themodule 300 of FIG. 22 . FIG. 24 is a side perspective diagram of themodule 300 of FIGS. 22 and 23 without the cage 315. FIG. 25 is aperspective diagram of a portion of the module of FIGS. 22-24 . Themodule 300, like the module 200, can be part of or inserted in a networkelement.

The module 300 is configured to receive the plug 320. The module 300includes a main PCB 305, and a cage 320 secured thereto. The cage 320 isconfigured to hold the plug 320 in place. The module 300 also includes amanifold 360. The manifold 360 includes a first manifold flow path 365and a second manifold flow path 366. While a single manifold 360 isillustrated, each of the flow paths can also be formed from separatemanifolds. The manifold flow paths are configured to form an inlet flowpath and an outlet flow path for the cooling fluid.

The manifold 360 also includes a first manifold fluid connector 355 anda second manifold fluid connector 356. The first manifold fluidconnector 355 fluidly connects the first plug fluid connector 335 to thefirst manifold flow path 365, and the second manifold fluid connector356 fluidly connects the second plug fluid connector 336 to the secondmanifold flow path 366. The fluid connection between the first manifoldfluid connector 355 and the first plug fluid connector 335 and the fluidconnection between the second manifold fluid connector 356 and thesecond plug fluid connector 336 can each be a quick disconnect. Thequick disconnect is configured such that the fluid connections can bequickly broken, while properly sealing each of the first manifold fluidconnector 355, the first plug fluid connector 335, the second manifoldfluid connector 356, and the second plug fluid connector 336 to preventleakage from both the integrated coldplate 330 and the manifold 360during removal of the plug 320 from the module 300.

Referring to FIG. 24 , the module 300 also includes a main PCB connector390 that is configured to receive and form an electrical connection withthe plug connector 329, thus, connecting the circuitry 322 of the plug320 to the main PCB 305 of the module 300.

In embodiments, the fluid connection between the first manifold fluidconnector 355 and the first plug fluid connector 335 and the fluidconnection between the second manifold fluid connector 356 and thesecond plug fluid connector 336 can each be a blind mate. The blind matetherebetween can be figured to act as a guide pin such that thereceiving of the first and second plug fluid connectors 335 and 336 bythe first and second manifold fluid connectors 355 and 356 can align theplug 330 in the cage 315 such that the electrical connection between themain PCB connector 390 and the plug connector 329 is properly formed.

FIG. 26 is a side perspective diagram of an embodiment of the module 300without the cage 315. In the embodiment illustrated, electricalconnector 390 includes the manifold fluid connectors. In the embodimentillustrated, the electrical connector 390 includes the female portion ofthe quick disconnect that receives the male first plug fluid connector335 and the second plug fluid connector 336. In the embodimentillustrated, the electrical connector 390, including the manifold fluidconnectors, are integrally formed as a unitary body, such as a singleplastic mold or closely-aligned thermally-connected-mated componentsforming integral connections. Forming the electrical connector 390,including the manifold fluid connectors, as a unitary body provides adirect thermal path from the electrical connection to the cooling fluidat the manifold fluid connectors. This direct thermal path can allow theelectrical connection to be cooled more effectively during the operationof the plug 320. In some embodiments, the unitary body includes coolingpaths that bring the cooling fluid closer to the electrical connectionfor further cooling thereof. These paths can be part of or separate fromthe cooling path to the coldplate 330.

Again, the manifold fluid connectors can act as guide pins for aligningthe electrical connection. With a unitary body making both theelectrical and fluid connections with the plug, the electrical connector390 can improve such alignment and can improve tolerancing between theelectrical and fluid connectors.

FIG. 27 is a schematic diagram of the connection zones of the module ofFIGS. 22-26 . In embodiments, the module 300 includes leak mitigation380. The leak mitigation 380 is positioned between the fluid couplingzone 382, such as the fluid connections, and the electrical connectionzone 384, such as the electrical connection between the electricalconnector 390 and the plug connector 329. The leak mitigation 380 can beintegrated into one or more components 302 of the module 300, such asthe manifold 360, the electrical connector 390, the plug 320, thecoldplate 330, the cage 315, and the like. In the embodiment shown inFIG. 26 , the leak mitigation 380 is positioned at an upper surface ofthe unitary body of the electrical connector 390, below the fluidconnection point. The leak mitigation 380 can be one or more of adepression in a surface, such as a channel, an absorbent materialpositioned below the fluid connection, such as above a surface of theelectrical connector 390 or in a channel of the electrical connector 390(as shown in FIG. 26 ), and the like. The leak mitigation 380 can beconnected to, integrated in, or integrally formed with the component302. Note, while described here with reference to the integratedapproach, the leak mitigation 380 can be used with the riding coldplate,the common coldplate, etc.

FIG. 28 is a schematic diagram of a leak mitigation system 370 for themodule 300. Referring to FIG. 28 , in some embodiments, the leakmitigation 380 is a sleeve. The leak mitigation system 370 includes thesleeve 380, a sensor 372, and a connector 374. The sleeve 380 surroundsa liquid connection 376 of the module 300, such as the liquid connectionbetween the first manifold fluid connector 355 and the first plug fluidconnector 335 and the liquid connection between the second manifoldfluid connector 356 and the second plug fluid connector 336. The sleeve380 includes an absorbent material, at least on the interior thereof,that collects discharged liquid, can include channels for guiding anyleaks away from electrical connections, and the like.

The sensor 374 is connected to one of the PCBs 378 in the module 300 viaone or more connectors 374. In the embodiment illustrated, the sensor374 is one of a humidity sensor and a vapor sensor positioned within thesleeve, such as within the absorbent material. Upon detection of apredetermined amount of humidity/vapor/moisture, one or more forms ofleak mitigation, such as closing valves to prevent the flow of thecoolant to the location of the leak and shutting down affected plugs 320to prevent overheating of the plugs 320, raising an alarm to a networkmanagement system (NMS), element management system (EMS), etc., and thelike, can be performed.

In some embodiments, the sensor is a pressure sensor that is used todetect a pressure drop in one of the various fluid flow paths throughoutthe system, which can be indicative of a leak. Upon detection of theleak, the one or more forms of leak mitigation can be employed.

In embodiments, multiple types of leak mitigation are usedsimultaneously. In some embodiments, multiple forms of leak mitigationoverlap to form layers of mitigation. For example, the plug 330 caninclude leak mitigation that overlaps with leak mitigation included atthe manifold 360 or electrical connector 390 when the plug is installed.Upon disconnect, the two forms of leak mitigation provide separate leakmitigation for the plug 330 and the manifold 360, respectively.

Sled Module

FIGS. 29 and 30 are perspective diagrams of an embodiment of a sledmodule 400. The sled module 400 includes a body 410, sled card guides412, one or more sled cards 420, main fluid manifold 460. The main fluidmanifold 460 includes a first main manifold fluid connector 455 and asecond main manifold fluid connector 456. The first main manifold fluidconnector 455 and the second main manifold fluid connector 456 fluidlyconnect to a sled card 420, such as to sled card fluid connectors, forproviding integrated cooling therein. The main manifold fluid connectorsand the sled card fluid connectors are configured to provide an inletand an outlet for the cooling fluid from the manifold 460. These fluidconnectors can include blind mate quick disconnects.

As described herein, a sled card 420 is a pluggable module that can pluginto the sled module 400, etc. to support some functionality in excessof optical interconnect. For example, the sled module 400 and each sledcard 420 can support switching, routing, storage, computing, etc. Alsoof note, the sled card 420 itself can support pluggable modules. So, thesled card 420 is itself a pluggable module that supports other pluggablemodules, e.g., pluggable optics.

The sled cards 420 are configured to receive pluggable optics modules,such as plugs 320. In embodiments, each sled card includes the manifold360, the first manifold fluid connector 355, the second manifold fluidconnector 356, the first manifold flow path 365, the second manifoldflow path 366, the electrical connector 390 of FIGS. 22-25 , the leakmitigation 380 of FIGS. 26 and 27 , and the leak mitigation system 370of FIG. 28 integrated therein. In some embodiments, the sled cards 420include the electrical connector 390 with the unitary body, includingthe integrated manifold fluid connectors, and leak mitigation 380 ofFIG. 26 . In these embodiments, the first and second manifold flow paths365 and 366 fluidly connect to the first and second main manifold fluidconnectors 455 and 456, respectively.

Example System

FIGS. 31-34 are perspective diagrams of an embodiment of a system 500,502 supporting 36 or 54 pluggable module 320, such as QSFP-DD sizedmodules. As described herein, the present disclosure integratesliquid-cooling technology into the pluggable module 10, 320. In variousembodiments, the pluggable module 320 can be QSFP's, QSFP-DD's, OSFP's,and CFP2's, but the concept applies equally to electrical-based modulesas well as other types of optical modules. The conventional techniquefor cooling pluggable optical modules is to use a sliding heatsink(air-cooled) on top of a pluggable module. The interface between thepluggable module and the sliding heatsink is a dry contact. The dryinterface results in a large thermal resistance that causes a largetemperature difference between the pluggable module (heat source) andthe heatsink.

With the emergence of 400 GbE technologies, the sliding interface is amajor barrier to cooling these high-power plugs. Additionally,air-cooling technologies are no longer sufficient to dissipate the heatgenerated by these high-power pluggable modules. The present disclosureeliminates the sliding thermal interface between the heat source and theheat sink, and also employs liquid-cooling or two-phase coolingcapabilities to remove large amounts of heat from the pluggable opticsin a very compact amount of space.

As described herein, this technology is made possible by replacing thetop case of the pluggable module 320, for example, a QSFP-DD module,with the integrated coldplate 330. No change is made to the internalcomponent arrangement and circuitry of the pluggable module 320. Theliquid fluid path within the module is shown in FIG. 21 . The moduleconnection to the main fluid line is made possible by two driplessquick-disconnects, namely the first plug fluid connector 335 and thesecond plug fluid connector 336.

Liquid lines 507 could be part of a closed-loop cooling system (i.e.,pump-reservoir-heat exchanger) or an open-loop cooling system, i.e.,facility coolant distribution manifold box. The design is compatiblewith multiple manufacturing processes. The liquid-cooled top case can bemade using metal 3D printing as a one-piece part. It can also be made asa two-piece part, body and lid that utilize cast, forged, metal 3Dprinted, or machined parts, and the same with the coldplate 230. The lidcan then be soldered/brazed to the main body to enclose the fluid flowpath. It can also be made by attaching a liquid flow tube 232 into agrooved base as shown in FIG. 10 . Attachment could be made by brazing,soldering, or any other thermally conductive methods.

The present approach has many advantages. With the elimination of thesliding thermal interface and taking advantage of the liquid-coolingand/or phase-change capacity for heat removal, this technology willallow for much higher plug powers. The cooling mechanism is integratedwithin the pluggable module 320, therefore no need for additionalheatsinking or cages that accommodate heatsink attachments.

The present approach is compatible with hybrid cooling systems(partially air/liquid cooled) with both side-to-side and front-to-backairflow designs. For side-to-side airflow designs, the air gets hotterfrom one plug to another resulting in less cooling ability as airtravels; in comparison, front-to-end airflow designs takes up space onthe faceplate. This solution allows improved performance of side-to-sideairflow designs.

36 plugs can be fit in 1Rack Unit 508 with plenty of room available onthe faceplate for airflow openings 509 to air-cool the rest of thecomponents inside the system (i.e., front-to-back airflow arrangement).

Using flyover cables 510, 54 plugs can be fit in 1 Rack Unit faceplate511 in the system 502 with side-to-side airflow arrangement as well as afully liquid-cooled system.

In a hybrid cooling system (partially air/liquid cooled), using flyovercables, the pluggable modules can be set up in horizontal 512 and/orvertical 513 orientations. This will bring about flexibility in thearrangement of pluggable ports and airflow openings on a faceplate 514to achieve the most efficient cooling solution on the air-cooledcomponents inside the system. The flyover cables allow signals to godirectly to the module resulting in less resistive connections thatimprove signal loss for high-speed signals, relative to prior artsolutions where the electrical connector sits on PCBs.

The liquid-cooled pluggable module 320 does not require additionalheatsinking. Therefore, the conventional cages can be eliminated and anextruded aluminum faceplate 515 can be used to accommodate 54 QSFP-sizedmodules in 1 rack unit. A guidance mechanism will be required for thepluggable modules to keep the electrical and liquid connectors in place.Unlike regular pluggables with heat sinks, there is no cage that sits onthe PCB; rather, the pluggables of this disclosure will mate directly tothe PCB and not require a cage, and therefore requires a guidancemechanism that help guide the plug to mate to the connector made.

This is the ultimate in cooling performance for a pluggable opticalmodule. Current heatsinking methods are proving increasingly difficultwith increases in the plug power. The present disclosure provides thecooling required for future high power pluggable optical modules (400GbE-800 GbE and beyond). The present disclosure enables extremefaceplate density (port count) with pluggable modules, in particularplugs with high power, high performance, and small form factor such aslong-reach coherent optics. Also, there is no need for additionalheatsinking. 36 plugs can be fit in 1RU with plenty of room available onthe faceplate for airflow openings to air-cool the rest of thecomponents inside the box. Using flyover cables, 54 plugs can be fit in1RU faceplate in a side-to-side airflow arrangement as well as a fullyliquid-cooled system.

Experimental Results

Liquid cooling is desirable because thermal properties of liquids arefar superior than those of air (i.e., orders of magnitudes better).Convective heat transfer is a product of “fluid flow properties” and“the space/surface available for cooling.” Product design is at thelimit of available space preventing additional heatsinks, requiring anew approach. With superior thermal properties of liquids, the spacerequired for cooling can be minimized, which opens the door for morecompact architectures.

Two liquid coldplates (i.e., liquid heatsink) were designed andprototyped, aiming to cool CFP2-DCO (400 GbE-32 W) (coldplate in FIG. 14) and QSFP-DD (400 GbE-18 W) (coldplates in FIGS. 7-11 ). The coldplateswere designed with the following objectives i) keep the optics belowtheir temperature ratings at elevated ambient temperatures, ii) smalldimensions (e.g., coldplates do not exceed the cage footprint), and iii)capable of cooling the modules with minimal liquid flow/pressuresupplied. For example, FIG. 15 includes 36xQSFP and their coldplates ina 1RU shelf. For a CFP2 form factor, the implementation is thin enoughto fit two CFP2s belly-to-belly in 1RU height.

A test setup was built to investigate the cooling performance of theliquid coldplates when used with working pluggable optics. A CFP2-DCOtest board was used as the test platform with an optical loop-backCFP2-DCO dissipating ˜23.7 W. An air-cooling benchmark test wasconducted for comparison purposes. To create an ideal air-coolingcondition, a 120 mm fan was placed on top of the CFP2 heatsink with airdirectly impacting on the heatsink surface (i.e., far beyond coolingprovided in real applications). The CFP2-DCO air-cooling test resultedin a CFP2 temperature 19.5° C. above ambient (i.e., ΔT=19.5° C.). TheCFP2 heatsink was removed and replaced by the 3 mm thick coldplate. Theliquid used for testing was a commercial propylene glycol solutionwidely used for electronics cooling. Three different flowrates weretested 1 mL/s, 1.5 mL/s, and 2 mL/s (i.e., liquid speed: 0.12 m/s to0.24 m/s) and the module temperatures were recorded (see FIG. 35 , whichis a graph illustrating temperature for different flowrates for theliquid cooling using a coldplate and for air cooling with a heat sink).Taking the lowest flowrate tested (i.e., 1 mL/s-0.12 m/s), the 3mm-thick liquid coldplate still outperforms the ideal air-coolingcondition with a 10 mm heatsink attached to a 120 mm fan.

Common Riding Liquid Coldplate

FIGS. 36 and 37 are various diagrams of a module 600 that supportsmultiple pluggable modules 10 with a common coldplate 630. FIG. 36 is aperspective diagram of the module 600, and FIG. 37 is a side perspectivediagram of the module 600 of FIG. 36 . The riding liquid coldplatedescribed herein includes the coldplate 230 per pluggable module 10. Inanother embodiment ufilizing the same concepts described herein, it ispossible to have the common coldplate 630 shared across multiplepluggable modules 10. The common coldplate 630 operates in the samemanner as the coldplate 230 and the integrated coldplate 330, excepthaving a larger surface area. Here, in the example of FIG. 36 , thereare 18 pluggable modules 10 in a row and 36 totals because there are tworows, and two common coldplates 630. The advantages of the commoncoldplate 630 are reduced liquid connectors relative to the coldplate230. However, the common coldplate 630 does not cool as well as thecoldplate 330 as the dry interface between the plug and the coldplateexists in this solution.

FIG. 37 illustrates thermal contact between a single pluggable module 10and the riding coldplate 230 or the common coldplate 630. Again, both ofthe coldplates 230, 630 are riding in the sense the pluggable module 10is removable. Here, there is a thermal contact 640 which can be thecoating on the coldplate 230, 630. It is diagonally tilted upward at theentry so as to prevent snagging when the pluggable module 10 isinserted. Once inserted, the pluggable module 10's housing on the topmakes contact with the thermal contact 640, and the cage can include aspring-type gasket 660 to bias the pluggable module 10 to the thermalcontact 640. The bias force can be tuned.

Advantages

The present disclosure provides significant temperature reduction ofIntegrated Tunable Laser Assemblies (ITLA) including nano-ITLA (nITLA)including at the nose of the plug especially in side-to-side airflowconfiguration. The integrated liquid cooled plug brings the cooling outto the nITLA even if it sits beyond the faceplate. This would reduce thetouch temperature of the case above the nITLA.

This approach allows delivery of more power to the plug: Temperaturerise at the electrical connector (i.e., power contact point) is alimiting factor for delivering power to recent/future high-rate plugs(400 G-800 G). Liquid-cooling could help to keep the power contacts cooland increase the power capacity.

For an integrated coldplate 330, the liquid and electrical connectionscan be integrated in a common body. The plug would connect to bothelectrical and liquid line at the same time. Liquid quick-disconnectswould work as alignment pins. This would help control tolerancing issuesbetween the liquid and electrical connections.

Also, the present disclosure provides extraction force improvement: Oneexisting problem is that dry-contact, which is associated with someextraction force. Although a low extraction force is desired (specifiedas a limit in the MSA), it is a choke-point in the thermal design:thermally, it is desirable to have high contact pressure, but this isnot possible because it is directly correlated with high extractionforce. The integrated liquid cooled pluggable allows elimination of theportion of the extraction force associated with heat sink contactpressure (while of course adding the quick connect force). The new quickconnect force is—in its optimization—a force less-than-or-equal-to theeliminated heat-sink-contact force. The quick connect force is fixed andconsistent and predictable among various solutions. It does not scalewith pressure or plug surface area or surface quality or finish type.

Upgradeable Hybrid Air/Liquid Cooling of Eletcro-Optical Systems withQuick Disconnects

The present disclosure pertains to an electro-optical communicationsplatform normally cooled with high velocity air. An alternate liquidcooling solution that is described here includes conceived and designedas a drop-in replacement for all or part of a conduction andconvection-based cooling system.

FIGS. 38-39 are various perspective diagrams of another network element700 platform. FIG. 38 is a front perspective diagram of the networkelement 700 platform, and FIG. 39 is a rear perspective diagram of thenetwork element 700 platform. The network element 700 is a shelf basedelectro-optical transmission system that includes slot-based circuitpacks 702 on its front side that plug into a backplane 704 for controlvia connectors to a set of control modules. The network element 700contains fan units 706 on its rear which plug into the rear of the samebackplane and receive control signals from the same control modules.

The circuit packs 702 (also known as modules, line modules, line cards,cards, plugs, blades, etc.) plug into a slot in a shelf 708 and connectto the backplane 704 to receive power and control signals. In anair-based cooling solution, each optical or electrical component with ahigh-power dissipation density is cooled with finned heatsinks, heatpipes, and/or vapor chamber assemblies. Devices in such systems arebecoming denser and producing more heat. We are at the limits of beingable to cool highly heat dissipative devices with the fans 706. On thenear horizon, we have high power devices (over 1000 W) and plugs (over30-35 W) that are beyond the capability of forced cooled convection inthe space available. This will necessitate liquid cooling.

The fans 706 have evolved to cool greater loads with revolutions perminute (RPM), noise, and power increasing to unreasonable levels. We arecurrently beyond the point where high-power air-cooled devices can existin a user-friendly environment free of ear protection.

Replacing all or part of the air-cooled system with liquid cooledcomponents allows the cooling of hotter devices and reduction in fanspeed to reduce the system noise to a workable level.

FIGS. 40-41 are perspective diagrams of a circuit pack 702 without avapor chamber 712 (FIG. 40 ) to show QSFP-DD heatsinks 713 and with thevapor chamber 712 (FIG. 41 ). A cover 710 protects the components belowfrom damage during insertion/extraction.

FIGS. 42-43 are perspective diagrams of the circuit pack 702 without avapor chamber 712 (FIG. 42 ) to show liquid cooling tubes of a liquidcooling harness 722 for liquid distribution going to cold blocks (notshown) to replace the QSFP-DD heatsinks 713 and with the vapor chamber712 (FIG. 43 ).

FIG. 44 is a diagram of the backplane 706 and the liquid cooling harness722 without other components in the shelf 708. FIG. 45 is a frontperspective diagram and FIG. 46 is a rear perspective diagram of acoolant distribution manifold box 730 connected to the liquid coolingsupport harness that can drop onto the circuit pack 702 with thebackplane 706 removed for illustration purposes.

Liquid cooling of devices in the circuit pack 702 is achievable byimplementing a system containing a coolant distribution manifold box 730or cooling line adjacent to the shelf 708, a connection of hoses fromthe harness mounted manifolds to a rear of a shelf mounted verticalinlet and return manifolds containing quick disconnect connections, andvarious manifolds and tubes to cold plates on the circuit pack 702 to becooled that mate to the quick disconnects on the vertical manifold 732.

Connecting a circuit pack 702 in a slot to a rear mounted manifold ismade possible by making a series of holes in the backplane 706 (see FIG.44 with the shelf 708 hidden) which can be used for air flow or passingof quick disconnect fittings from a circuit pack 702 to a rear manifold732. Quick disconnects on the liquid cooled circuit can simultaneouslyor concurrently connect with electrical connectors on the backplane 706.

Although the network element 700 shown shows horizontal circuit packs702 on the front and a vertical manifold 732 on the rear, thisconfiguration could reverse such that there are vertical circuit packson the front and horizontal manifolds on the rear. The coolantdistribution manifold box 730 includes various input and output ports onthe vertical manifold 732 to connect to multiple liquid coolingharnesses 722 and single input/output ports 734 on the rear of thecoolant distribution manifold box 730.

In an embodiment, a liquid cooling section can be provisioned byremoving one of the rear airflow cooling modules and replacing it with aliquid cooling distribution/harness module. Since the coolingrequirements drastically drop by liquid cooling the hottest componentsthe need for all the previous cooling unit modules is changed. Airdistribution to all un-liquid cooled components can still be considered,but a large portion of the air-cooling system can be removed or turneddown in speed, to reduce noise and power consumption.

The rear vertical manifold 732 can be attached to the rear of thebackplane 706 or could be a removable unit within the size of theremoved air-cooling module.

FIGS. 47-50 are various perspective diagrams of a liquid cooling harness722. FIG. 47 is a top perspective diagram of the liquid cooling harness722. FIG. 48 is a bottom perspective diagram of the liquid coolingharness 722. FIG. 49 is a bottom perspective diagram of the liquidcooling harness 722 with cooling lines 740 removed. FIG. 50 is a topdiagram of the liquid cooling harness 722 with cooling lines 740connected to the rear manifold 732.

Furthermore, the ability to upgrade an existing electro-optical circuitpack 702 from air cooled to hybrid liquid and air cooled is achievablewith a single drop-in liquid cooled harness 750. All items on theharness attach floatingly to a support plate 752 that suspends coldplates 754 and 756 below it. All cold plates 754 and 756 can bepre-assembled and connected fluidly to a series of tubes and manifoldsin the cooling lines 740 for equal fluid flow distribution. The attachedcold plates 754 can be suspended with shoulder type screws that allowthe components play in a vertical position with respect to the circuitboard they are attached to. Once in place the cold plates 754 can makecontact with hot devices on the main circuit board by various means.

Front mounted cold plates 756 can be preloaded with springs, or leafsprings can be added during assembly, to push down into QSFP-DD cageopenings. Once a QSFP-DD pluggable device is plugged in a dry-contact ismaintained with the force from the pre-attached or subsequently-attachedsprings. Also, a cold plate 754 for an ASIC is shown with 4 corner mountscrews 758. The mount screws 758 can be shoulder screw type, with alarge flat head pushing down on a spring in each corner. The other endof the spring pushes the cold plate toward the AS IC to apply a goodforce to this fixed ASIC. A thermal interface pad can be used on fixedtype devices.

The fluid paths and devices shown on the liquid cooling harness 722 arejust an example. The system can be extended in any direction toencompass more items on the circuit board.

Furthermore, a second harness could be attached to the opposite side ofthe main circuit card to cool items on the opposite side of the board. Amanual disconnect could be used to join a top based harness to a bottombased harness. Alternatively, the harness 722 can be configured to loopdown to the secondary side with coolant lines for secondary cooling.

FIGS. 51-52 are a front perspective view (FIG. 51 ) and a rearperspective view of the coolant distribution manifold box 730. Eachhorizontal position requires an inlet quick disconnect and an outletquick disconnect, and hence there is one vertical inlet manifold and onevertical outlet manifold. The unit will also have on its rearconnections to an inlet hose and outlet hose, shown as large diametertubes on the rear. It is also advantageous to have an automatic ormanual air bleeder valve at the top of the system, so that air does notbuild up in the system. Bleeder valves at the bottom can be used tobleed or drain fluid from the system. The manifold supply and return mayeach include a port for an air bleed valve and a port for a drain. Thecoolant distribution manifold box 730 may also include an array ofguidance pins, each of which is used to align a circuit pack moduleduring its insertion to a position that is within an acceptance windowof the quick disconnects.

The present disclosure allows removably pluggable liquid cooling ofelectrical and optical components in a backplane-controlled system, suchthat quick disconnects pass through the backplane and interconnectsimultaneously with electrical and optical connections. It also allowsthe conversion of an air-cooled system to a liquid cooled system with adrop in liquid cooled harness containing its quick disconnect andencompassing all liquid cooled sections.

The platform can include a mix of liquid-cooled circuit packs andair-cooled circuit packs within one hybrid system. There can also bedouble blind mating of liquid connections for in-service modulereplacement: insert a rear liquid cooling unit into existing circuitpacks or insert a circuit pack into existing rear liquid cooling units.

The rear module can optionally have one or more of the following withinthe module: fans, liquid interfaces, active fabric circuits, or coolantdistribution manifold box components (e.g., pumps, radiators, etc.).There might or might not be a backplane present.

The airflow path can have 90-degree bends; not limited to front-to-backairflow. There can be redundant scenarios whereby a front card caninterface with two or more rear liquid modules.

The vertical manifold may be integrated with the shelf assembly, or itmay exist as a module which optionally attaches to the shelf assembly.In the latter option, the manifold module may be added to the shelfassembly as needed, or in a specific implementation, it can replace afan module.

This system allows air cooled cooling devices like heatsinks, vaporchambers, and loop heat pipes to be replaced on the same circuit cardwith a drop-in liquid cool harness encompassing manifolds and liquiddistribution and containing a quick disconnect connection to a rearmounted vertical manifold distribution system. The drop-in harness ispre-filled, and pressure tested with liquid before swapping it into acircuit pack.

Conclusion

It will be appreciated that some embodiments described herein mayinclude or utilize one or more generic or specialized processors (“oneor more processors”) such as microprocessors; Central Processing Units(CPUs); Digital Signal Processors (DSPs): customized processors such asNetwork Processors (NPs) or Network Processing Units (NPUs), GraphicsProcessing Units (GPUs), or the like; Field-Programmable Gate Arrays(FPGAs); and the like along with unique stored program instructions(including both software and firmware) for control thereof to implement,in conjunction with certain non-processor circuits, some, most, or allof the functions of the methods and/or systems described herein.Alternatively, some or all functions may be implemented by a statemachine that has no stored program instructions, or in one or moreApplication-Specific Integrated Circuits (ASICs), in which each functionor some combinations of certain of the functions are implemented ascustom logic or circuitry. Of course, a combination of theaforementioned approaches may be used. For some of the embodimentsdescribed herein, a corresponding device in hardware and optionally withsoftware, firmware, and a combination thereof can be referred to as“circuitry configured to,” “logic configured to,” etc. perform a set ofoperations, steps, methods, processes, algorithms, functions,techniques, etc. on digital and/or analog signals as described hereinfor the various embodiments.

Moreover, some embodiments may include a non-transitorycomputer-readable medium having instructions stored thereon forprogramming a computer, server, appliance, device, processor, circuit,etc. to perform functions as described and claimed herein. Examples ofsuch non-transitory computer-readable medium include, but are notlimited to, a hard disk, an optical storage device, a magnetic storagedevice, a Read-Only Memory (ROM), a Programmable ROM (PROM), an ErasablePROM (EPROM), an Electrically EPROM (EEPROM), Flash memory, and thelike. When stored in the non-transitory computer-readable medium,software can include instructions executable by a processor or device(e.g., any type of programmable circuitry or logic) that, in response tosuch execution, cause a processor or the device to perform a set ofoperations, steps, methods, processes, algorithms, functions,techniques, etc. as described herein for the various embodiments.

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure, arecontemplated thereby, and are intended to be covered by the followingclaims.

What is claimed is:
 1. A hybrid air/liquid-cooled network elementcomprising: a backplane defining a plurality of holes; one or moreprinted circuit boards disposed on one side of the backplane; aplurality of fan units coupled to an opposite side of the backplane andadapted to circulate an air flow to components disposed on the one ormore printed circuit boards through certain of the plurality of holesdefined by the backplane; and one or more coolant distribution manifoldboxes coupled to the opposite side of the backplane adjacent toplurality of fan units and adapted to circulate a coolant fluid flow toone or more cold plates disposed adjacent to components disposed on theone or more printed circuit boards through other of the plurality ofholes defined by the backplane; wherein the one or more coolantdistribution manifold boxes each have height and width dimensionssubstantially the same as each of the plurality of fan units.
 2. Thehybrid air/liquid-cooled network element of claim 1, wherein theplurality of fan units are arranged in rows and columns along theopposite side of the backplane with the air flow delivered orthogonallyto the one or more printed circuit boards.
 3. The hybridair/liquid-cooled network element of claim 2, wherein each of the one ormore coolant distribution manifold boxes is arranged among the pluralityof fan units in one of the rows and columns.
 4. The hybridair/liquid-cooled network element of claim 1, wherein each of the one ormore coolant distribution manifold boxes comprises one or more manifoldsadapted to couple one or more coolant lines coupled to the networkelement to quick disconnects coupled to the one or more cold plates. 5.The hybrid air/liquid-cooled network element of claim 1, wherein the oneor more printed circuit boards are each disposed in a module.
 6. Thehybrid air/liquid-cooled network element of claim 5, further comprisinga shelf, wherein the backplane is coupled to the shelf and the moduleassociated with each of the one or more printed circuit boards isdisposed in the shelf.
 7. A coolant distribution manifold box for use ina hybrid air/liquid-cooled network element, the coolant distributionmanifold box comprising: one or more manifolds adapted to be coupled toone or more coolant lines coupled to the network element; and quickdisconnects adapted to be coupled to one or more cold plates disposedadjacent to components disposed on one or more printed circuit boardsdisposed within the network element; wherein the coolant distributionmanifold box has height and width dimensions substantially the same as afan unit adapted to be coupled to the network element.
 8. The coolantdistribution manifold box of claim 7, wherein a plurality of fan unitsare adapted to be arranged in rows and columns along a backplane of thenetwork element with an air flow delivered orthogonally to the one ormore printed circuit boards.
 9. The coolant distribution manifold box ofclaim 8, wherein the coolant distribution manifold box is adapted to becoupled to the backplane of the network element and arranged among theplurality of fan units in one of the rows and columns.
 10. The coolantdistribution manifold box of claim 7, wherein the one or more printedcircuit boards are each disposed in a module.
 11. The coolantdistribution manifold box of claim 10, wherein the module associatedwith each of the one or more printed circuit boards is disposed in ashelf of the network element.
 12. A method for converting an air-coolednetwork element to a hybrid air/liquid-cooled network element, themethod comprising: given a backplane defining a plurality of holes, oneor more printed circuit boards disposed on one side of the backplane,and a plurality of fan units coupled to an opposite side of thebackplane and adapted to circulate an air flow to components disposed onthe one or more printed circuit boards through certain of the pluralityof holes defined by the backplane, removing one or more of the pluralityof fan units from the opposite side of the backplane; and coupling oneor more coolant distribution manifold boxes to the opposite side of thebackplane in one or more spaces associated with the removed one or moreof the plurality of fan units and adjacent to the remaining plurality offan units, wherein the one or more coolant distribution manifold boxesare adapted to circulate a coolant fluid flow to one or more cold platesdisposed adjacent to components disposed on the one or more printedcircuit boards through the certain of the plurality of holes defined bythe backplane; wherein the one or more coolant distribution manifoldboxes each have height and width dimensions substantially the same aseach of the plurality of fan units.
 13. The method of claim 12, whereinthe plurality of fan units are arranged in rows and columns along theopposite side of the backplane with the air flow delivered orthogonallyto the one or more printed circuit boards.
 14. The method of claim 13,wherein each of the one or more coolant distribution manifold boxes isarranged among the plurality of fan units in one of the rows and columnsafter coupling.
 15. The method of claim 12, wherein each of the one ormore coolant distribution manifold boxes comprises one or more manifoldsadapted to couple one or more coolant lines coupled to the networkelement to quick disconnects coupled to the one or more cold plates. 16.The method of claim 12, wherein the one or more printed circuit boardsare each disposed in a module.
 17. The method of claim 12, wherein themodule associated with each of the one or more printed circuit boards isdisposed in a shelf of the network element.
 18. The method of claim 12,wherein the one or one or more printed circuit boards comprise aplurality of printed circuit boards and the method further comprises aircooling some of the plurality of printed circuit boards using theplurality of fan units and liquid cooling some of the plurality ofprinted circuit boards using the one or more coolant distributionmanifold boxes.